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ABSTRACT 

The aim of this study is (a) to investigate the usefulness of Bloom's revised taxonomy (RBT) 
for classification of standards, (b) to examine the differences and similarities between pre¬ 
service teachers' and in-service teachers' classification of the same standards and (c) to 
determine which standards are vague and broad. The 45 standards, in the Turkish 10 th 
Grade Physics Syllabus, were categorized by the 16 participants, who were divided into 
two groups. The first group included eight pre-service physics and the second group 
included eight in-service physics teachers, in Turkey. Firstly, each participant classified 
the standards using RBT individually, then, they classified the standards with their group. 
We compared their all classification of standards. The usefulness of Revised Bloom 
Taxonomy for classification of standards, the differences between individually 
classification and the groups' and, differences between the pre-service and in- service 
teachers groups' classifications, (c) the standards which are broad and vague were 
discussed. 
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Introduction 

What students are supposed to know, what they are expected to learn in each 
subject, what students should have learned/acquired by the end of a subject or a 
course and how the students should attain knowledge are defined as standards 
(Tyler, 1949; Bloom, 1956; Anderson and Krathwohl, 2001). Standards guide the 
instructors in selecting/developing the content, developing an instructional 
strategy, and assessment instruments. Standards are not only necessary for 
instruction design, but also necessary for instructors, students, curriculum 
supervisors, and administrators. They help the students to know what they are 
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supposed to learn. Curriculum supervisors and administrators will be able to 
develop quality assessment instruments based on these standards and implement 
quality management systems. Standards are used as a tool by schools and 
teachers to focus on learning outcome. Standards should help teachers to monitor 
achievement and to develop programs that improve student learning and can be 
used by teachers as a reference point for learning programs, for planning teaching 
process and for assessing. According to Skilbeck (2007); 

"Without clearly formulated and precise standards it is difficult to 
prepare valid tests and other forms of assessment. This point is of great 
significance to the student, who is entitled to know, in advance, at least 
the general outlines of what he is expected to do in order to perform 
adequately. The existence of clear and detailed standards is a help to 
both teachers and students in judging progress towards attainment of a 
goal. Having a clear idea of standards enables the teacher in the 
classroom to pick out of the lightly structured teaching-learning 
situations which are now so common, (projects, visits, case-studies, 
discussions, etc.), elements of potential significance, and to build on 
these." 

As Skilbeck (2007) said that the existence of clear and detailed 
objectives/standards is a help to both teachers and students in judging progress 
towards attainment of a goal. However, interpretation of standards is based on 
teachers' judgments, and, if the standards interpreted similarly or differently by 
teachers, the outcomes will be similar or different. Because, how the standards 
are interpreted by teachers/instructors is important for planning of teaching and 
for learning outcomes. So, it is important that teachers agree on their 
interpretation of standards and teachers have to interpret the standards similarly 
to get equivalent in their own country or region. But, researcher state that many 
standards are broad and vague (Patton and Trainor, 2003; Popham, 2003; Wiggins 
and McTighe, 2005; Luft, Brown, and Slutherin 2007; Nasstrom, 2009) and this 
may affect the interpretation of standards. Then, how we may determine the 
differences and similarities of interpretation of standards, and, which standards 
are broad and vague is an important question. 

According to some researchers (O'Neil and Murphy, 2010; Nasstrom, 2009), 
standards can be classified and interpreted in a variety of ways and a taxonomy 
may be useful. Also, Anderson and Krathwohl (2001), and Krathwohl (2002) 
stated that the Taxonomies is a scheme for classifying educational goals, 
objectives, and, most recently, standards. It provides an organizational structure 
that gives a commonly understood meaning to objectives classified in one of its 
categories, thereby enhancing communication. By classifying/interpreting the 
standards into cells of a taxonomy, whose aim is to make standards clearly 
understandable, we can see the structure of the standards. Thus, we can see 
differences and similarities among judges such as teachers, pre-service teachers 
and experts while classifying the standards. 

There are many taxonomies, such as Bloom's original taxonomy (1956), 
Guilford's taxonomy (1967), SOLO taxonomy (Biggs and Collis, 1982), Porter's 
taxonomy (Porter and Smithson, 2001), Bloom's Revised Taxonomy-Revision of 
original- (Anderson and Krathwohl, 2001), Dettmer taxonomy (2006), and 
Marzano's new taxonomy (Marzano and Kendall, 2007), can be used to interpret 
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standards. But, in this study the RBT (Revised Bloom's taxonomy) was selected 
and used as a tool for classifying standards. 

Why Did We Use Revised Bloom's Taxonomy (RBT)? 

Some of the reasons for using the Revised Bloom's Taxonomy in this study 
include: 

1. RBT is a classification of learning objectives within education that 
educators set for students (Krathwohl, 202). It is important to establish 
intended learning outcomes in professor/student interactions so that both 
parties understand the purpose of the interactions. Developing intended 
learning outcomes using RBT helps faculty to design and implement 
appropriate assessment tasks, measures, and instruments. Having 
intended learning outcomes based on RBT helps to ensure that instruction 
and assessment are appropriately aligned with the intended outcomes. 
The RBT is designed for developing learning objectives, teaching and 
assessment among these main components of an educational system 
(Nasstrom, 2009). 

2. Nasstrom (2009) compared two taxonomies, RBT and Porter's, in order to 
assess their usefulness for alignment analysis based on Hauenstein's five 
rules and on inter-judge reliability, she found out that Bloom's revised 
taxonomy empirically was more useful than Porter's model. Bloom's 
revised taxonomy is useful for interpretation of standards and taxonomy 
is found to be the most useful model for classification of standards. 

3. In another study where standards in chemistry and mathematic were 
categorized in two different types of models, Bloom's revised taxonomy 
was found to interpret the standards more unambiguously than a model 
with topics-based categories and it is the most useful classification tool 
today and it is a useful tool for interpreting the standards (Nasstrom and 
Henriksson 2008; Nasstrom, 2009). 

4. An (2008) stated that Bloom's Taxonomy is widely used and has been 
concerned with the international stage, and it's the mostly used and well- 
known taxonomy in educational settings. The use of this taxonomy is not 
limited to the US, it has been used all over the world. 

5. Krathwohl (2002) stated that RBT has many merits on the usage as a tool 
to analyze the learning objectives, it could serve as a common language 
about learning objectives, and RBT provides teachers and educators with 
a common frame of reference that clarifies various types of learning 
outcomes. 

6. RBT has been developed for all academic subjects and allows comparisons 
of standards from different subjects and teachers need a framework to 
help them to make sense of objectives and organize them so that they are 
clearly understood and fairly easy to implement (Anderson and 
Krathwohl, 2001). The RBT has been applied to both pre-school education 
and higher education and to all types of academic subjects. 

7. When pre-service teachers learn about writing instructional objectives, 
they are typically asked to categorize or label each one according the 
Bloom's level of cognitive complexity (Gronlund, 2004). Borich, Tombari 
and Tombari (2004) found that the taxonomy benefits teachers in planning 
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their lessons. It helps teachers to focus on the outcomes, specifically 
instructional objectives, which they want their students to attain as a 
result of instruction. 

8. Finally, the RBT, well known taxonomy in Turkey and it is taught to pre¬ 
service teachers in their education by instructors and it is used by in- 
service teachers at school and educators training students to become 
teachers often refer to Bloom’s Taxonomy during each aspect of the 
instructional cycle, from planning to assessing instruction (An, 2008). 

Literature Review 

There are many studies about RBT; from classification of questions of any 
subject to produce a quality exam paper via or based on the RBT. Also, there are 
different studies from to enhance students' learning in biology (Crowe, Dirks, and 
Wenderoth, 2008), to challenge students' thinking (Vrchota, 2004), from to 
develop critical thinking (Athanassiou, McNett, and Harvey, 2003), to develop a 
planning tool to differentiate curriculum (Noble, 2004), from to interpret 
standards (Nasstrom, 2009), and to teach students about plagiarism (Vosen and 
Fink, 2008). 

Indeed, the importance of categorizing standards with RBT was addressed by 
Anderson and Krathwohl, (2001) and Krathwohl (2002) stated that using the RBT 
table (see Figure 1) to classify or to interpret the objectives, activities, and 
assessments provides a clear, concise, visual representation of a particular course 
or unit and indicate that the teachers can use the RBT for classifying 
standards/objectives. Bloom's Taxonomy provides an established, useful and 
comprehensive framework for identifying/writing/interpreting instructional 
objectives (Gronlund, 2004). It's said that RBT can be used to categorize 
objectives, but person(s) doing the classification should make correct inferences. 
And individuals may disagree on the accurate classification of an objective. 

However, there is lack of studies that examine the usefulness of RBT for 
classification/interpretation of standards/objectives differences and investigation 
similarities among judges' classification of standards, and determine the 
disagreement about the correct classification of a standard. Accordingly in this 
study, while searching for literature review we found a few studies, like Nasstrom 
(2009), -who studied differences and similarities between teachers and 
assessment experts when they interpreted standards by means of the taxonomy, 
She investigated the differences between the teachers and the assessment experts 
when interpreting the standards of mathematic syllabus. The teachers utilized 
the categories in RBT to a larger extent, multi-categorized to a lesser extent, and 
had lower levels of inter- and intra-judge consistency than the assessment 
experts. The assessment experts were more consistent in their interpretation of 
standards. According to her conclusions, agreement on interpretation of standards 
had to increase, especially for teachers. 

Due to the lack of studies, this study should be carried out. And, this study 
focuses on (a) the usefulness of RBT for classifying the standards, (b) the 
similarities and differences between pre-service teachers and in-service teachers' 
classifications and (c) the vague and broad standards of 10th grade physics 
syllabus, in Turkey, during the pre- and in- service teachers classifying the 
standards, by using the RBT. The main questions of the study are as follows: 

1. Is the RBT useful in the categorization of standards? 
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2. What are differences and similarities while classifying standards? 

a) What are differences and similarities among individuals, groups (4 judges) 
and final (8 judges) classification of pre-service teachers? 

b) What are differences and similarities among individuals, groups (4 judges) 
and final (8 judges) classification of in-service teachers ? 

c) What are differences and similarities between pre-service teachers' and 
in-service teachers' classification of standards? 

3. Which standards are vague and broad? 

To answer the first question, we took into consideration the Hauenstein's 
(1998) five rules. According to Hauenstein, a taxonomy should (a) be applicable; 
(b) be totally inclusive, i.e. all standards can be categorized; (c) have mutually 
exclusive categories, i.e. unambiguously categorize one standard into only one 
category; (d) follow a consistent principle of order; and (e) use the terms in 
categories and subcategories that are representative of those used in the field. To 
answer second question Percent Agreement, Krippendorff s alpha, Cohen's kappa 
and, Emphasis Index were used to determine the differences and similarities 
between/among pre- and in- service teachers' classification and to calculate intra¬ 
rater, inter-rater consistency. And, to answer third question the agreement level 
was used. 

Method 

Two groups of pre-service and in-service teachers classified the same 
standards (10th grade physics syllabus) with Bloom's Revised Taxonomy under 
similar conditions. The participants of these two groups were compared regarding 
inter- and intra-judge agreement. The list of verbs appropriate for the levels of 
Bloom's taxonomy (the cognitive domain, see figure 2) was also used in clarifying 
the process of classification. The study was performed using the document 
analysis method within the framework of qualitative research. Document analysis 
is the process of collecting records and documents on the relevant subject and 
examination by coding them according to a system and certain norms (Creswell 
and Clark, 2011). 

Revised Bloom's Taxonomy(RBT) 

Bloom's original cognitive taxonomy (1956) was revised and modified in order 
to remove adapt it for 21. century World (An, 2008). The important change of new 
taxonomy is that it is two-dimensional instead of one-dimensional. The verb and 
noun forms are separated from each other into two dimension: Knowledge 
Dimension and Cognitive Process Dimension (Krathwohl and Anderson, 2001; 
Amer, 2006). 

The knowledge dimension is a different type of knowledge: Factual 
(knowledge of terminology, knowledge of specific details and elements), conceptual 
(knowledge of classifications and categories, knowledge of principles and 
generalizations, knowledge of theories, models, and structures), procedural 
(knowledge of subject-specific skills and algorithms, knowledge of subject-specific 
techniques and methods, knowledge of criteria for determining when to use 
appropriate procedures), and metacognitive knowledge (strategic knowledge, 
knowledge about cognitive tasks, including appropriate contextual and 
conditional knowledge, self-knowledge). This dimension focuses on content as 
types of knowledge, and according to Krathwohl and Anderson (2001), to lie along 
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a continuum, from concrete in factual knowledge to abstract in metacognitive 
knowledge. 

The cognitive processes dimension is intended to provide a comprehensive set 
of classification for students' cognitive processes that are included in objectives 
(Krathwohl and Anderson, 2001). The categories in this dimension are remember, 
understand, apply, analyze, evaluate and create. This dimension represents a 
continuum of increasing cognitive complexity, from lower order thinking skills 
(remember) to higher order thinking skills (create). 

Bloom's revised taxonomy table consist of two-dimension and compose of 24 
cells (see Table 1, adapted from Anderson and Krathwohl, 2001, p.46). The rows 
in the taxonomy table represent the four categories of the knowledge dimension 
and the columns the six categories of the cognitive process dimension. Columns 
are used to represent the verbs in the objective and rows are used to represent the 
nouns in the objective. One standard will thereby be categorized according to the 
two dimensions and placed in the corresponding cell(s) in the taxonomy table. 
Finally, each objective can be classified in one cell or more than one cell. (Amer, 
2006; Krathwohl, 2002). But in this study each objective can be classified just one 
cell. 


Table 1. Revised bloom's taxonomy table 


Remember 

Understand 

Apply 

Analyze 

Evaluate 

Create 

Recognizing 

Recalling 

Interpreting 

Comparing 

Exemplifying 

Classifying 

Summarizing 

inferring 

Explaining 

Executing 

Implementing 

Differentiating 

Organizing 

Attributing 

Checking 

Critiquing 

Generating 

Planning 

Producing 

Factual Cell 1 

Cell 2 

Cell 3 

Cell 4 

Cell 5 

Cell 6 

knowledge 






Conceptual Cell 7 

Cell 8 

Cell 9 

Cell 10 

Cell 11 

Cell 12 

knowledge 






Procedural Cell 13 

Cell 14 

Cell 15 

Cell 16 

Cell 17 

Cell 18 

knowledge 






Metacognitive Cell 19 

Cell 20 

Cell 21 

Cell 22 

Cell 23 

Cell 24 

knowledge 







How the RBT Can Be Used? 

Standards must fall under one of the four categories under the knowledge 
dimension, and under one of the six categories of the cognitive process dimension. 
Use the noun in the objective to determine what is being learned: factual, 
conceptual, procedural, or meta-cognitive knowledge (the rows in the taxonomy 
table). The verb used in the learning objective will determine which cognitive 
process dimension column the objective falls under: remember, understand, apply, 
analyze, evaluate, and create (the columns in the taxonomy table). The objective 
is then placed in the corresponding cell in the taxonomy table, where the 
knowledge and cognitive process dimension intersect. For example, it can be seen 
in the Figure 1. How a pre/in-service teachers placed one objectives into cell. 











(50 INTERNATIONAL JOURNAL OF ENVIRONMENTAL a SCIENCE EDUCATION 


11303 


Objective 11. The student explain the interaction between charged particles. 


Noun: the interaction between charged particles. 

s 

Knowledge Dimension 

A. Factual 
.Conceptual 

(the interaction between charged particles) 

C. Procedural 

D. 


Verb: explain 



Cognitive Process Dimension 

1. Remember 

2. Understand 
/(explain) 

3. Apply 

4. Analyze 

5- kx&s&Usi 

6. Create 




Understand / 
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Evaluate 

Create 
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o. 

S 
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O / 

/ = 
f e 

vk 

j 
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Explaining 

•I 

1 

X 

Is] 

I 

"S 

DilVcrcntialin 

i 

o 

Attributing 

1 

s 

2 

O 

M 

3 

•-C 

C 

o 

Generating 

J 

1 

_3 

Factual 

knowledge 

Cell 1 

\ 7 





Conceptual 

knowledge 


*CeIl(S * 

Objective 11 





Procedural 

knowledge 



Celll 1 




knowledge 




Celll6 


Cell24 


Figure 1 . How an objective is classified in the taxonomy table? 

Participants: The Pre-Service and In-Service Teachers 

Eight pre-service and eight in-service teachers participated in this study. The 
pre-service teachers were at the their last semester (10th semester) to graduate 
from department of secondary science and mathematics education, faculty of 
education. The department aims particularly to equip students with the up-to- 
date knowledge and practical skills required for secondary school mathematics 
and science teachers. An important objective of the department is to provide its 
graduates with relevant contemporary information, training, and prerequisite 
skills to enable them to guide their students to cope with the challenges of the 
21st century. Their graduate program consist of content of physics (i.e mechanic, 
optic, electricity and magnetism), and Educational Psychology, Curriculum 
Development and Instruction, Theories and Approaches in Teaching and 
Learning of Science, Measurement and Evaluation, Computer Applications, 
Instructional Technology and Material Development, Classroom Management, 
Turkish Educational System and School Management, Practice Teaching at 
Schools, Research Projects in Science. After graduating from faculty of education, 
the prospective teachers attain a national exam, KPSS, to be in-service teachers. 
The KPSS consist of 80 multiple choose questions that deal with their subject (i.e 
Educational psychology, Curriculum development etc.) and 50 multiple questions 
includes content of physics as average a pre-service/prospective teachers should 
correct 70/80 of the first section and 35/50 of the second section to be teacher at 
national high school. 

The in-service teachers working at high school. They teach students from 9th 
to 12th grade. They have at least 3 years of experiences. 
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As a result, we can say that both pre-service teachers and in-service teachers 
are similar with curriculum, syllabus, Bloom Taxonomy and the 10th grade 
physics syllabus objectives and content. 

The standards/objectives 

In Turkey, a curriculum is defined by a policy document for each educational 
level. The aim of the physics curriculum (9th-12th grade) is to improve scientific 
literacy and the scientific process abilities of students. The curriculum of physics 
which include 9 th and 10 th grades deals with the basic terms of physics. It is within 
the frame of this program that students be able to attain both functional and 
conceptual knowledge and apply them to different and new situations. The 11 th 
and 12 th grade education curriculum reveals the same concepts more-detailed and 
linking with other physical concepts. Hence the aim of this program is to prepare 
the students for university or for a choice of physical-related carrier. Therefore, it 
is intended that 11 th and 12 th grade students be able to continue their studies in 
a field related to physics alongside with learning it. 

A syllabus in Turkey is a policy document for one subject, for example 10th 
grade physics syllabus (TTKB, 2013), at a specific educational level, containing 
one type of standards. The standards of the 10th grade physics syllabus is that 
students be able to interpret basics about optics, magnetism, electricity, force, 
pressure which are components of physics without having to deal with detailed 
mathematical knowledge. There are 4 subjects and 45 standards in 10th grade 
physics syllabus (see Table 2). 


Table 2. 10th grade physics syllabus (TTKB) 


Unit no 

Unit name 

The number of 
objectives 

Class hours 

Time 

Percentage 

1 

Pressure 

4 

10 

13,9 

2 

Magnetism, electricity 

13 

22 

30,6 

3 

Wave 

11 

16 

22,2 

4 

Optic 

17 

24 

33,3 


Total 

45 

72 

100 


Procedure and Data Gathering 

The study was planned in the spring semester of 2013-2014. By meeting an 
academician who has experience and studies on Bloom taxonomy and revised 
taxonomy (Kocakaya and Gonen, 2010) both the aim, importance and context were 
fully explained. After this deciding to participate 13 pre-service teachers were 
inquired whether to take part in the study or not. And the process started after 
the approval of 8 of them, but we waited for the 2014-2015 spring term so as to 
apply Bloom taxonomy. 

The detailed stages of the process are as follows: 

1. Being an expert on taxonomy associate professor fully explained how to 
categorize the standards into RBT, and the original taxonomy in 4 weeks 
and 3 hours per each week. (Week 1) 
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2. The instructor categorized the objectives of 9 th grade syllabus with RBT 
by working with pre-service teachers in 2 weeks so that they gain 
experience. Verb list (figure 2) was used while categorizing. (Week 3). 


Revised Bloom's Taxonomy of the Cognitive Domain 

Cognitive Level 

Sample Verbs to Use in Writing Intended Student Learning Outcomes 

Remembering 

Define 

Duplicate 

Identify 

List 

Name 

Recall 

Recognize 

Reproduce 

Retrieve 

Tell 

Understanding 

Calculate 

Categorize 

Clarify 

Classify 

Compare 

Conclude 

Contrast 

Describe 

Discuss 

Exemplify 

Expand 

Explain 

Identify 

Illustrate 

Infer 

Interpret 

Locate 

Match 

Outline 

Paraphrase 

Predict 

Report 

Restate 

Summarize 

Translate 

Applying 

Carry out 
Classify 

Demonstrate 

Execute 

Illustrate 

Implement 

Practice 

Solve 

Use 

Utilize 

Analyzing 

Appraise 

Attribute 

Compare 

Contrast 

Deconstruct 

Detect 

Differentiate 

Discriminate 

Distinguish 

Examine 

Formulate 

Infer 

Integrate 

Organize 

Parse 

Relate 

Select 

Sequence 

Structure 

Test 

Evaluating 

Appraise 

Check 

Coordinate 

Critique 

Defend 

Detect 

Dispute 

Judge 

Monitor 

Prioritize 

Rate 

Reconstruct 

Select 

Support 

Verify 

Creating 

Change 

Combine 

Compile 

Compose 

Construct 

Create 

Design 

Formulate 

Generate 

Hypothesize 

Improve 

Invent 

Plan 

Predict 

Produce 


Figure 2. The verb list. 


3. 10th grade physics syllabus objectives (45 objectives, see table 2), The RBT 
tables (see Table 1) and verb list (Figure 2) were given to pre-service 
teachers and they were asked to categorize each objective individually, 
(see Figure 1). (Week 4). Each participant was coded as PI, P2 so on. 

4. They were divided into 2 groups each of which included 4 pre-service 
teachers in week 4. The groups were selected from different classes. Each 
of them was asked to re-group the 45 standards and they were recorded 
as GAl and GA2. 

5. 8 teachers clubbed together (final) and categorized the 45 standards after 
an evaluation and discussion and coded it as FA. 

6. Therefore all of the 45 standards were initially categorized individually 
(i.e P1,P2...) next in four (i.e GAl,..) and eventually in eight (FA) by pre¬ 
service teachers. After gathering information from the pre-service 
teachers was completed, the process of gathering information from in 
service teachers started. 

7. Visiting the schools nearby the teachers willing to participate were 
selected. While selecting it was paid the utmost care that teachers be 
experienced and have knowledge on taxonomy. 

8. Having selected 8 teachers each being from a different school each teacher 
was given a file of memorial and descriptive information about RBT and 
information about how to classify the standards of 9 th grade. Moreover 
they were given 10th grade objectives, table and verb list and were asked 
to categorize these in a week. 
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9. The categorizations of 8 teachers were taken individually and were coded 
as T1,T2 but data gathering process was ceased as it was summer holiday. 

10. In September 2015 teachers were divided into 2 groups of four. These 
groups came together on different days and locations and did 
categorizations. The former was coded as GT1 and the latter as GT2. 

11. After 12 days all of the 8 teachers came together and made a final 
categorization to be called (FT) by in-service teacher, as done by pre¬ 
service teachers previously. (See Figure 3) 



Figure 3. The process of classification of objectives by pre-service teachers, P. The same 
process carried out for in-service teachers, as well. 


Data Analysis 

To the end of the study 22 worksheets were gathered; 16 of which individual 
(8 of in-service and 8 of pre-service teachers'), 4 of which that of groups (2 of pre¬ 
service and 2 of in-services teachers') and 2 of which belonging to final studies. 

How the data were analyzed and which statistical methods were used and 
why the method were used explained in detail as follows: 

In this study (1) Average Pairwise Percent Agreement , (2) Krippendorff's 
alpha (a), (3) Cohen’s kappa , (4) The frequencies and percentages and, (5) 
Emphasis Index were used to determine the differences and similarities 
between/among pre- and in- service teachers by calculating intra-rater, inter-rater 
consistency, both two and more than two raters. And the agreement levels were 
calculated to determine how many participant classified the standards into cell. 

1. In both individual, group and final categorizations the frequency and 
percentage of the cells both used and not-used in the taxonomy chart will 
be given thereby enabling us to make comparisons both within and 
between groups. (See figure 3). The statistical analysis of inter- and intra¬ 
judge consistency for panels as wholes is based on how the standards are 
distributed in the taxonomy table for all judges in each panel. The 
percentage of the total number of classifications of all standards is 
presented in a taxonomy table for each panel on each occasion. 

2. Average Pairwise Percent Agreement will make the number of standards, 
similarity percentage of in and inter-groups done by teachers available. 

3. Average Pairwise Cohen's Kappa (Inter-rater consistency) will make 
available the consistency rate of in-group of teachers. And eventually the 
consistency rate of both panels will appear. Cohen’s kappa coefficient is a 
statistic which measures inter-rater for qualitative (categorical) items. It 
is generally thought to be a more robust measure than simple percent 
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agreement calculation, since k takes into account the agreement occurring 
by chance. It works when assessing the agreement between two raters. In 
this study to evaluate the comparisons of GAl and GA2 , GT1 and GT2 , 
and finally of Al and FT1 this scale will be employed. 

4. Krippendorffs Alpha (nominal) will let us make inter/intra rater 

calculations. Because of multiple judges and the data on nominal level, 
Krippendorff's alpha (a) are used in this study. Hayes and Krippendorff 
(2007) have gone through why this analyze would be used in this study. 
Krippendorff s alpha (a) is a reliability coefficient developed to measure 
the agreement among observers, coders, judges, raters, or measuring 
instruments drawing distinctions among typically unstructured 
phenomena or assign computable values to them. It is general in that it 
can be used regardless of the number of observers, levels of measurement, 
sample sizes, and presence or absence of missing data, a emerged in 
content analysis but is widely applicable wherever two or more methods 
of generating data are applied to the same set of objects, units of analysis, 
or items and the question is how much the resulting data can be trusted 
to represent something real. Its extension to many observers is stated in 
analysis of variance terms. Thus, a is in good company. Alpha also allows 
for reliability data with missing categories or scale points, a frequent 
reality that none of the reviewed measures has been able to cope with 
(Hayes and Krippendorff, 2007). Landis and Koch (1977) gave the 
following table for interpreting values. 


Table 3. Interpreting values of ft] 


Interpretation 

K 

Poor agreement 

< 0 

Slight agreement 

0.01 - 0.20 

Fair agreement 

0.21 - 0.40 

Moderate agreement 

0.41 - 0.60 

Substantial agreement 

0.61 - 0.80 

Almost perfect agreement 

0.81 - 1.00 


In this study Krippendorff s scales will be used in explanation and discussion 
first for the individual studies of both pre and in-service teachers and then for 
G1+G2+F categorizations. 

4. The frequencies and percentages of standards with agreement levels will 
be given. Therefore, that whichever objectives fits where accordingly with 
its agreement level and the agreement level of both pre and in-service 
teachers will be compared. 

5. 6.The distribution of standards in the different cells in the taxonomy table 
for each occasion is compared to the distribution of the other occasion and 
as a measure of how similar the two distributions are the emphasis index 
is used. This index has been used by Porter (2002) for comparing the 
distribution of standards with the distribution of assessment items in 
alignment analyses, and used by Nasstrom (2009) to compare the teachers 
and experts’ classifications of standards. The emphasis index is: 
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2 

where x is the proportion of the total number of categorized standards in each 
cell inthe taxonomy table for Panel 1 or Occasion 1 and y is the corresponding 
proportion for Panel 2 or Occasion 2. When E = 1, the distributions are the same 
and emphasize the same cells in the taxonomy table. E = 0 means that the 
distributions are completely different. Webb (2002) used a similar index for 
balance between standards and assessment items, and according to him index 
values of at least 0.70 indicate an acceptable level. Index values between 0.60 and 
0.70 are, according to Webb, indicating an only weakly acceptable level. 

Results 

What is the frequency and percentage of the cells used in all categorizations 
presented in the Figure 4. 

What is the number of the cells both used and not used during the study, the 
frequency of usage of cells, the number of the cells used each time by pre-service 
teachers, the number of the cells both used and not used by both pre and in-service 
teachers presented in the Table 4. 


Table 4. The number of ceils for categorizations 



Used 

cells 

Non 

used 

The number of 
cell which is 
most used 

Non used cells 
for 

categorization by 
groups 

non used cells by Both 
pre and in-service 
teachers 

Used cells 
for all 
occasions 

Non used any 
categorization 

Used cells 
by both 
group and 
individual 

Pre Ind 

23 

1 

73/360 (No:10) 









% 20,2 






G1 

9 

15 

17/45 (No:10) 

Pre-service 








% 37,7 

9 cells 


5 cells 



G2 

13 

11 

12/45 (No:16) % 

No: 


No: 






26,6 

3,5,6,12,13,14, 


2, 



F 

11 

13 

10/45 (No:10) % 

,20,21 


8,10,16,24 






22,2 






In Ind 

22 

2 

73/360(No:8) 


Both pre and in non 

10 cells 

1 cell 

5 cells 




% 20,2 


used 

No: 

No:19 

No: 2,8,10, 

G1 

10 

14 

9/45 (No:8 and 

In-service 

3 cells 

1,2, 


16, and 24 




No:10) 

6 cells 

No:, 6,20,21 

8,9,10,11,1 



G2 

12 

12 

14/45 (No:8) 

No: 6,7, ,18, 


4,15,16,24 



F 

13 

11 

11/45 (No:8) 

,20,21,22 







Remember 

Understand 

Apply 

Analyze 

Evaluate 

Create 

1 Individually 

3 

o 

Final 

Individually 

3 

o 

Final 

Individually 

3 

o 

Final 

Individually 

3 

o 

Final 

1 Individually 

3 

o 

Final 

1 Individually 

3 

o 

Final 
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Figure 4. The percentage and frequencies of the cells used. .Distribution Jn per cent, of 
total classifications of all standards on each occasion .Individually. G1_ G2 and Final. 


Note: while there were 8*45-360 in individual classifications there happened 
to be 45 classifications each in groups and final . And this becomes the answer for 
the difference in percentages. 

In individual classifications the pre-service teachers used 24 cells while 
leaving one out. The most frequently used cell is conceptual knowledge dimension 
(74 times). The in-service teachers used 22 cells and leaving 2 ones out. The most 
frequently used cell by teachers is that of procedural knowledge dimension (61 
times). None of the pre-and in- service teachers used the cell remember 
metacognitive knowledge (No: 19), while all the other 23 cells were used by at least 
one pre- or in-service teachers on at least one occasion. 

In Gl meetings by locating all of the objectives into 9 of the 24 cells the pre¬ 
service teachers used the conceptual-analyze cell most, on the other hand the in- 
service teachers positioned all the objectives into 10 cells out of 24 and the concept- 
understand and concept-analyze cell became them so frequently used ones with 
the rate of 9 times. 
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In G2 meetings, pre-service teachers settled the objectives into 13 cells out of 
24 and used procedural-analyze cell 12 times on the other hand the in-service 
teachers put the objectives into 13 out of 24 cells and used concept-understand 
cell most frequently. 

In final meetings the pre-service teachers put the objectives into 11 of the 24 
cells and used procedural-analyze cell 10 times. On the other hand, the in-service 
teachers settled all objectives into 13 cells out of 24 and used concept-understand 
cell 11 times. 

When we take classifications with groups (Gl, G2, F) into consideration the 
pre-service teachers used only 15 cells leaving out 9 and in-service teachers used 
18 cells leaving out 6 cells. 3 of these cells are the same for both parties. That is, 
number 6, 21 and 20 have not been employed by neither party. 

In all classifications both individually and in groups, the number of the cells 
used by pre-service in all circumstances is 5 (2,8,10,16,24) while in-service 
teachers used 10 ones namely (1,2,8,9,10,11,14,15,16,24) and finally 5 
(2,8,10,16,24) and number 19 has not been used by neither group. 

What is consistency rate between the classifications (Gl, G2, F) of pre and in- 
service teachers (Krippendorff s Alpha statistics) presented in Table 5. 


Table 5. KrippendorlTs Alpha statistics _ 

Krippendorffs Alpha 



Pre 

0.114 

Individually 

In 

0.057 


Pre 

0.217 

G1+G2+F 

In 

0.193 


Pre 

0.168 

Both individually and G1+G2+F 

In 

0.089 

Both teacher and pre 

Individually 

0.087 

Both teacher and pre 

Group (G1+G2+F) 

0.168 


The consistency among the individual classifications of the pre-service 
teachers is 0,112 while that of the in-service teachers is 0,055. Notably the 
consistency among that of the pre-service teachers is higher than that of the in- 
service ones. In the classification of the teachers as final group of 8 and 2 groups 
(G1,G2) and individuals the consistency rate is 0,211 and its 0.187 for in-service 
teachers. Again the consistency of the pre-service teachers is higher than that of 
the in-service teachers. While the consistency among all classifications done by 
pre-service teachers (individual, +G1+G2+F) is 0,166 that of the in-service 
teachers is 0,087. The consistency of pre-service teachers’ classifications is higher 
than that of the in-service teachers. The consistence between the pre-service and 
in-service teachers is 0.086 while that of the groups is 0.165. The consistency of 
group classifications is higher than that of the individuals. 

What is the Average Pairwise Percent Agreement (%) Average Pairwise 
Cohen's Kappa of objectives in both individual and as group classifications 
presented in Table 6. 
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Table 6. Average Pairwise Percent Agreement _ Average Pairwise Cohens Kappa of 
objectives in both individual and as group classifications. 




Average Pairwise Percent 

Average Pairwise Cohen's 



Agreement (%) 

Kappa 

Individually 

Pre 

21.0 

0.116 


In 

15.4 

0.067 

G1 and G2 

Pre 

22.2 

0.104 


In 

24.4 

0.273 

G1 and Final 

Pre 

31.1 

0.192 


In 

35.5 

0.139 

G2 and final 

Pre 

44.4 

0.362 


In 

26.6 

0.169 

G1+G2+F 

Pre 

32.5 

0.219 


In 

28.8 

0.194 

Both individually 

Pre 

26.5 

0.170 

and G1+G2+F 

In 

18.5 

0.095 

Both teacher and 

Individually 

17.8 

0.092 

pre 

Both teacher and 

Group 

26.8 

0.171 

pre 

(G1+G2+F) 




As is clear in the table the accordance between pre-service teachers’ GAl and 
final classification is 31,1 while that of the GA2 and final is 44,4 and that of the 
GAl, GA2 is 22,2. In other words the number of the objectives put into the same 
cell is 14 while that of the GA2 and final is 20 and 10 for GAl and GA2. 

As it is evident from the table the consistency rate of the teachers in GT1 and 
final classifications is 26,667 and 24,4 for GT1 and GT2. That is to say the number 
of the objectives established in the same cell is 16 in both GT1 and final meetings 
while it is 12 for final and GT2 and eventually 11 for GT1-GT2. 

The consistency percentage between pre and in-service teachers' individual 
classification is 17.815 (Kappa= 0.092) while that of the group classification is 
26.815 (Kappa=0.171). The consistency percentage among group classification is 
higher than that of the individuals. 

What is the number of the objectives put into the same cell in classifications 
and by how many people was it done presented in Table 7. 


Table 7. Agreement level for each standards_ 

In-service Teachers Pre-service Teachers 


1: No agreement 10112, 10234 

f: 2 
% 4,4 

2-3: Poor 3.10114, 3.10211, 3.10212 

2.10213, 3.10214, 2.10221 
2.10232, 2.10233, 3.10242 
2,2.10252, 3.10313,2.10312 
2.10314,3.10322,2,2.10331 
2.10332, 3.10333, 2.10341 
3,3.10411, 2.10412,3.10421 
3.10441, 3.10461, 2.10463, 
3.10464, 2.10471, 3.10491 
2.10493,2.104101,3.104103 
f: 30 
% 66,6 


f:0 

%0 


3.10112,3.10211,2.10212 
2.10213,2.10214,3.10221 
2.10232,3.10233,3.10234 
2.10241,3.10252,3.10311 
2.10312,2.10313,3.10314 
2,2.10321,2.10322, 3.10331 
2,2.10332,3.10343,2.10411 
3.10412,2.10421,2.10431 
2,2.10441,2.10451,3.10461 
3.10464,3,3.10471,3.10481 
3,3.10492,3,3.10493,3.104101 
f: 33 
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4-5: Moderate 

5.10111,4.10113,5.10231 

%73,3 

4. 10111,4.10113,4.10114 


5.10241,5.10251,4.10313 

5.10242, 4.10251,4.10323 


4.10321,4.10323,4.10431 

4.10333,4.10462,4.10463 


4.10451,4.10462,5.10492 

5.10491 


f: 12 

f: 10 


% 26,6 

%22 

6-7: Substantial 

6.10481 

6.10231 


f:1 

f:1 


%2,2 

% 2,2 

8: Perfect 

f: 0 

104103 


%0 

f:1 



%2,2 


Note: Each objective starts at 10. The number in front of 10 reveals that by 
how many people has the objective been put into the same cell. For instance 
5.10111 code shows that number 10111 objective has been put into the same cell 
by 5 people. 


Table 8. The frequencies and percentages of objectives with agreement levels 



P re-service T 

In-service T 

Similar 

1: 

f:0 

f: 2 

0 

No Agree 

0 

4,4 


2-3: 

f: 33 

f: 30 

22 

Poor A 

73,3 

66,6 


4-5: 

f: 10 

f: 12 

5 

Moderate A 

22 

26,6 


6-7: 

f:1 

f:1 

0 

Substantial A 

2,2 

2,2 


8: 

f:1 

f: 0 

0 

Perfect A 

2,2 

0 



Note: If one standard categorized in each cell by just 1 person (no agreement); 
2-3 persons (Poor); 4-5 persons (Moderate); 6-7 persons (Substantial); 8 persons 
(Perfect). 


As can be seen in the table the number of the objectives put into 8 different 
cells by 8 different teachers is 0. 33 objectives have been put into the same cell by 
2 or 3 teachers, 10 objectives to the same cell by 5 or 4 and consequently 1 objective 
into the same cell by 6 teachers. The number of the objectives put into the same 
cell by 8 teachers is 1. As in the table 2 objectives were put into the same cells by 
8 pre*service teachers while 30 objectives were put in the same cell by 2or 3 
teachers, 12 objectives into the same cell by 4 or 5 teachers and consequently 1 
objective was put in the same cell by 6 teachers. The number of the objectives put 
into the same cell by 8 teachers is 0. while 22 of the objectives where both pre-and 
in-service teachers have poor agreement upon are same 5 of the moderate 
agreement are same. While the number of the those with Substantial agreement 
is same, these objectives are different from each other. That is the objective put 
into the same cell have been put into different cells by 3 pre-service teachers. 
What is the Emphasis index among classifications presented in Table 9. 


Table 9. Emphasis index 
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Occasion 1 (x) 

Occasion 2 

(y) 

E 

Acceptable level* 

Pre-service 

Gl 

Individually 

0,40 

Weakly acceptable level 

teachers 

G2 

Individually 

0,80 

acceptable level 


Final 

Individually 

0,70 



Final 

Groupl 

0,95 

acceptable level 


Final 

Group2 

0,95 

acceptable level 


Groupl 

Group2 

0,90 

acceptable level 

In- 

Gl 

Individually 

0,80 

acceptable level 

Service 

G2 

Individually 

0,90 

acceptable level 

Teachers 

Final 

Individually 

0,65 



Final 

Groupl 

0,90 

acceptable level 


Final 

Group2 

0,90 

acceptable level 


Groupl 

Group2 

0,80 

acceptable level 

Between 

Pre-Final 

In-Final 

0,80 

acceptable level 


pre- and 

in-service 

teachers 


_ Pre-lndividually _ In-lndividually _ 0,65 Weakly acceptable level 

*According to Webb (2002). = ~* * 


In terms of Emphasis index, the E value is 0,70 between individual and final 
classifications, Gl-INDIVIDUAL is 0,95 INDIVIDUAL-G2 is 0,90. As seen in the 
table the E score of final group classifications of both pre and in-service teachers 
are than that of the individual ones. The E scores between the individual 
classifications of both pre and in-service teachers is 0,65 while that of the finals is 
0,80. With regards to E scores the rate of the group classification is higher than 
that of the individuals. This fact proves that classification with group is much 
more efficient. 


Discussion 

Usefulness of RBT 

It has been observed that both pre and in-service teachers have put the 
objectives into the taxonomy table. This case shows that RBT is inclusive in 
classification of objectives as Nasstrom (2009) stated. Whether the cells in the 
taxonomy table mutually exclusive or not we can look that if one standard is 
placed in only one category. In this study one standard is placed in only one 
category by pre or in service teachers while, in her study, the judges were allowed 
to multi-categorize, i.e. categorize one standard into more than one cell, and the 
judges utilized this possibility, But, in our/this study the judges (pre-service and 
in-service teachers) has to categorize just one standard into one cell. Therefore, it 
is possible to say that the categories in Bloom’s revised taxonomy are mutually 
exclusive. But in all classifications held in the process of study the researches have 
been present and done observations. In the classifications individuals have had 
troubles originating from the need of doing multi-categorization. Especially when 
in the case of two verbs of objectives they had trouble with which cell to put it and 
decision making has been left to them. For example, while the participants 
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classifying the standard "By using the magnetic field concept, the students explain 
attractive and repulsive forces between the magnets and analyzes the variables that 
this force is connected”, they were discussed whether it will be in cell of 
understand or analyze. 

When the level of inter- and intra-judge consistency are considered, 
Krispendorrf alpha percentages of agreement are quite low. This case proves that 
both pre and in-service teachers have differences in the classification of the 
objectives and that there is low rate similarity between group and individual 
classification. The reason may be objectives were not clear, plain and 
comprehensible and the objective composing principles were ignored could be 
count among the reasons. (Patton and Trainor, 2002; Popham, 2001; Wiggins and 
McTighe, 2005; Luft, Brown, and Slutherin2007; Nasstrom, 2009). This case can 
cause to teaching in different stages on the same subject by affecting the teaching 
techniques. When considering the alpha and kappa scores this situation can cause 
different teaching levels about the same level via affecting the teaching strategies. 
The individual alpha (0,086) and kappa (0,092) scores of pre-and in-service 
teachers’ individual classification and those of the group classifications alpha 
(0,165) kappa (0,171) suggests that the consistency rate of group classification is 
much higher than that of the individual classification. 

Taking applied classifications into consideration 23 cells from the 24 in the 
taxonomy table have been used at least by one teacher except for metacognitive 
knowledge-remember. This case shows that RBT table is efficient in the 
classification of the objectives. 

As a result, we can say that according to Hauenstein's (1998) rules Bloom's 
revised taxonomy is a useful tool for classifications of standards of physics 
syllabus in this study. 

Similarities and Differences Between Pre-Service and In-Service 
Teachers 

The pre- and in-service teachers categorized the standards almost the same 
into cells in the taxonomy table (see Figure 1). In the classifications both pre and 
in-service teachers have employed more cells in individual classifications than in 
group classifications. In individual classification pre-service teachers have put the 
45 objectives into 23 cells, in-service teachers put them into 22 cells while in the 
8 people-group classifications pre-service teachers put them into 11 and in-service 
teachers put in 13 cells. Pre-service teachers used conceptual-knowledge- 
understand cell most frequently while in-service teachers put into conceptual- 
knowledge-understand cell. This case suggests that in alignment of objectives 
cognitive process dimension differences were visible than those of the information 
dimension. This case can be due to the experiences of both pre and in-service 
teachers and the lack of verbal dimension of objectives. While neither in- nor pre¬ 
service teacher used lcell, they have used 5 cells in both individual and group 
classifications. 

Considering the consistency in the study (Krippendorffs Alpha, Percent 
Agreement and Cohen's kappa) its rate is low for both parties. However, all 
classifications of pre-service teachers are higher than either of the in-service 
teachers. In fact, in-service teachers were supposed to give higher rates when 
considering their experiences. On the other hand, the knowledge of pre-service 
teachers on taxonomy, objectives were still fresh and what is more they had been 
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involved with a study with 9 th grade and these must have affected the result 
scores. 

The Vague and Broad Standards 

The frequencies and percentages of objectives with agreement levels are 
considered, the pre- and in-service teachers categorized the same standards 
almost the same cells with same agreement levels. Teachers have been observed 
having poor agreement on more than half of the objectives (33 for pre and 30 for 
in service teachers and of 22 are the same) and moderated agreement almost on 
a quarter (10 for pre and 12 for in service teachers and of 5 are same) the number 
of the objectives on which both parties had higher rate of agreement is 1. This can 
be a result of objectives’ being comprehensive and vague. And having such low 
agreement ratio on the objectives can have negative effects on teaching processes. 

Taking Emphasis indexes into consideration it is clearly visible that the E 
scores of the pre-service teachers is higher than that of the in-service teachers. 
The E levels of both parties is comparatively higher than those of the individuals 
and finals. Still considering both parties’ individual and final classifications, the 
E score between two finals is 0,80 higher than that between two individual 
classifications which is 0,65. 

Limitation 

The multi-categorization of objectives was not made possible for this study. 
But, researchers have been present and done observations during the whole 
period of the study. In the classifications performed the performers have felt the 
need for putting the objectives into multi-categorization and thereby experienced 
some problems. Especially in the case of two verbs the performers have had 
discussions which cognitive process to put the objectives and the decision was left 
to them. 

The size of the samples in this study is quite small and this may have 
influenced the reliability negatively. However, a large number of judges also 
require a lot of resources as time, people and money, and therefore the level of 
acceptable reliability has to be weighed against the costs. 

The teachers and pre-service teachers in this study are not fully 
representative of pre-in service teachers in general. They were in the same city 
and the researchers could arrive them. If a random sample of teachers had 
participated, the levels of inter- and intra-judge consistency could be expected to 
have been even lower than for the teachers in this study. Also, all the judges in 
this study are assumed to be familiar with the standards. 

This study is also limited to standards in physics, while Bloom’s revised 
taxonomy was developed for all academic subjects. Therefore, it is possible that 
studies with standards from other academic subjects than physics can give 
different results concerning the degree of usefulness of Bloom’s revised taxonomy 
and the levels of inter- and intra-judge consistency. 

Conclusions 

How the standards are interpreted by teachers is important for planning of 
teaching and for learning outcomes. Because interpretation of standards is based 
on teachers’ judgments, and, if the standards interpreted similarly or differently 
by teachers, they will teach similarly or differently ways and then, the outcomes 
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will be similar or different. But seem in this study, the low levels of inter- and 
intra-judge consistency, especially for the in-service teachers, indicate 
considerable differences in their classifications of standards. Therefore, it is 
important to study and report inter- and intra-coder consistency when standards 
are interpreted. As Nasstrom (2009) said experience of interpreting standards 
seems to be an important qualification for teachers. So, to increase the teachers' 
qualification, a more extended training session might be one solution. 

One conclusion is that agreement in interpretation of standards has to 
increase, especially for teachers, to allow all students a fair chance to attain the 
same standards regardless of teachers and schools. The low levels of inter- and 
intra-judge consistency for the teachers may have negative effects on students’ 
learning, on fairness in assessments. 

To be able to recommend Bloom’s revised taxonomy for interpretation of 
standards in general, further studies in other academic subjects than physics are 
needed. There are many academic subjects but a starting-point might be in 
subjects that often have standardized assessments. 

One of the notable findings of this study is that one can know whether 
everyone comprehend the ambiguity of objectives in the same manner and this 
becomes possible through RBT. The agreement level for objectives (especially for 
43 of 45 objectives) in this study is rather low. High agreement ratio was observed 
only on two objectives. In other words, just one objective was put into the same 
cell by everyone involved. Thus with the RBT it can be found out if the objectives 
in a teaching program is understood in the same way by everyone. Moreover, 
putting the same objective into different cells may mean that the objective has 
vague meanings. The motives of this situation should especially be studied. 
Whether low-agreement originates from negligence of writing criteria for 
objectives or from any other reason should be studied also the fashion in which an 
objective is written must be clear for anyone. Because, well-written learning 
objectives can give students precise statements of what is expected of them and 
provide guidelines for assessing student progress. Our goal for students is 
learning and if students don’t know what they should be able to do at the end of 
class then it will be difficult for them to reach that goal. Clearly defined objectives 
form the foundation for selecting appropriate content, learning activities, and 
assessment measures. If objectives of the course are not clearly understood by 
both instructor and students, then the methods of assessment, which are 
supposed to indicate to both learner and instructor how effective the learning and 
teaching process has been, will be at best misleading, and, at worst, irrelevant or 
unfair. When a learning objective is well written and accurately describes what 
we want the participant to know or what knowledge to be gained then it will guide 
the instructor to properly developing and structuring the course. 

Eventually, in the classification of objectives as seen in this study, that 
teachers come together and make classifications with RBT in groups enables the 
objectives that lead teaching-learning processes to be effectively evaluated and 
contributes to assessment and planning as well. 
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